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Abstract — The recent work of Sommer, Feder and Shalvi 
presented a new family of codes called low density lattice 
codes (LDLC) that can be decoded efficiently and approach the 
capacity of the AWGN channel. A linear time iterative decoding 
scheme which is based on a message-passing formulation on a 
factor graph is given. 

In the current work we report our theoretical findings 
regarding the relation between the LDLC decoder and belief 
propagation. We show that the LDLC decoder is an instance 
of non-parametric belief propagation and further connect it to 
the Gaussian belief propagation algorithm. Our new results 
enable borrowing knowledge from the non-parametric and 
Gaussian belief propagation domains into the LDLC domain. 
Specifically, we give more general convergence conditions for 
convergence of the LDLC decoder (under the same assumptions 
of the original LDLC convergence analysis). We discuss how 
to extend the LDLC decoder from Latin square to full rank, 
non-square matrices. We propose an efficient construction of 
sparse generator matrix and its matching decoder. We report 
preliminary experimental results which show our decoder has 
comparable symbol to error rate compared to the original 
LDLC decoder. 

I. Introduction 

Lattice codes provide a continuous-alphabet encoding pro- 
cedure, in which integer-valued information bits are con- 
verted to positions in Euclidean space. Motivated by the 
success of low-density parity check (LDPC) codes [1], recent 
work by Sommer et al. [2] presented low density lattice 
codes (LDLC). Like LDPC codes, a LDLC code has a sparse 
decoding matrix which can be decoded efficiently using an 
iterative message-passing algorithm defined over a factor 
graph. In the original paper, the lattice codes were limited 
to Latin squares, and some theoretical results were proven 
for this special case. 

The non-parametric belief propagation (NBP) algorithm 
is an efficient method for approximated inference on con- 
tinuous graphical models. The NBP algorithm was originally 
introduced in [3], but has recently been rediscovered indepen- 
dently in several domains, among them compressive sensing 
[4], [5] and low density lattice decoding [2], demonstrating 
very good empirical performance in these systems. 

In this work, we investigate the theoretical relations be- 
tween the LDLC decoder and belief propagation, and show 
it is an instance of the NBP algorithm. This understanding 
has both theoretical and practical consequences. From the 



theory point of view we provide a cleaner and more standard 
derivation of the LDLC update rules, from the graphical 
models perspective. From the practical side we propose to 
use the considerable body of research that exists in the NBP 
domain to allow construction of efficient decoders. 

We further propose a new family of LDLC codes as well 
as a new LDLC decoder based on the NBP algorithm . 
By utilizing sparse generator matrices rather than the sparse 
parity check matrices used in the original LDLC work, 
we can obtain a more efficient encoder and decoder We 
introduce the theoretical foundations which are the basis of 
our new decoder and give preliminary experimental results 
which show our decoder has comparable performance to the 
LDLC decoder 

The structure of this paper is as follows. Section |ll] 
overviews LDLC codes, belief propagation on factor graph 
and the LDLC decoder algorithm. Section III rederive the 
original LDLC algorithm using standard graphical models 
terminology, and shows it is an instance of the NBP algo- 
rithm. Section |IV] presents a new family of LDLC codes as 
well as our novel decoder We further discuss the relation to 
the GaBP algorithm. In Section [V] we discuss convergence 
and give more general sufficient conditions for convergence, 
under the same assumptions used in the original LDLC 
work. Section VI brings preliminary experimental results 
of evaluating our NBP decoder vs. the LDLC decoder We 
conclude in Section VII. 

II. Background 
A. Lattices and low-density lattice codes 

An n-dimensional lattice A is defined by a generator matrix 
G of size n x n. The lattice consists of the discrete set of 
points X — {xi,X2, ■■■,Xn) S i?" with x — Gb, where b G Z"^ 
is the set of all possible integer vectors. 

A low-density lattice code (LDLC) is a lattice with a non- 
singular generator matrix G, for which H ~ G^^ is sparse. 
It is convenient to assume that det{H) = \/det{G) = 1. An 
{n, d) regular LDLC code has an H matrix with constant row 
and column degree d. In a latin square LDLC, the values of 
the d non-zero coefficients in each row and each column are 
some permutation of the values hi,h2, - ■ ■ , hd- 

We assume a linear channel with additive white Gaussian 
noise (AWGN). For a vector of integer- valued information b 



the transmitted codeword is a; = Gb, where G is the LDLC 
encoding matrix, and the received observation is y = x + w 
where w is a vector of i.i.d. AWGN with diagonal covariance 
a^I. The decoding problem is then to estimate b given the 
observation vector y; for the AWGN channel, the MMSE 
estimator is 

6* = arg min \ \y — Gb\\^ . (1) 

B. Factor graphs and belief propagation 

Factor graphs provide a convenient mechanism for repre- 
senting structure among random variables. Suppose a func- 
tion or distribution p{x) defined on a large set of variables 
1 factors into a collection of smaller functions 
p{x) = fs{xs), where each Xg is a vector composed of a 
smaller subset of the Xi. We represent this factorization as a 
bipartite graph with "factor nodes" fs and "variable nodes" 
Xi, where the neighbors Ts of fs are the variables in Xs, and 
the neighbors of Xi are the factor nodes which have Xi as an 
argument (fs such that Xi in Xg). For compactness, we use 
subscripts s,t to indicate factor nodes and i,j to indicate 
variable nodes, and will use x and Xg to indicate sets of 
variables, typically formed into a vector whose entries are 
the variables Xi which are in the set. 

The belief propagation (BP) or sum-product algorithm [6] 
is a popular technique for estimating the marginal probabili- 
ties of each of the variables Xi. BP follows a message-passing 
formulation, in which at each iteration r, every variable 
passes a message (denoted MJ'^) to its neighboring factors, 
and factors to their neighboring variables. These messages 
are given by the general form, 

M[+\x,) = f{x,) n M[,{xi), 
teri\s 

M:+\x,)^ f ,Ux,) H Mj,ixj)dx,. (2) 

Here we have included a "local factor" fi{xi) for each 
variable, to better parallel our development in the sequel. 
When the variables Xi take on only a finite number of values, 
the messages may be represented as vectors; the resulting 
algorithm has proven effective in many coding applications 
including low-density parity check (LDPC) codes [7]. In 
keeping with our focus on continuous-alphabet codes, how- 
ever, we will focus on implementations for continuous- valued 
random variables. 

1) Gaussian Belief Propagation: When the joint distri- 
bution p{x) is Gaussian, p{x) oc exp{— ^a;-'" + h^x}, the 
BP messages may also be compactly represented in the same 
form. Here we use the "information form" of the Gaussian 
distribution, A/'(x; /x, S) = N^^{h,J) where J = 
and h = J/i. In this case, the distribution's factors can 
always be written in a pairwise form, so that each function 
involves at most two variables Xi, xj, with fij{xi,Xj) = 
exp{- JjjXjXj}, j ^ i, and fi{xt) = exp{-i Jijxf + hiX^}. 

Gaussian BP (GaBP) then has messages that are also 
conveniently represented as information-form Gaussian dis- 



tributions. If s refers to factor /y , we have 

Mj+\x,)=J^-\p,^,,a,^,), 

= Jii+ ^ Qffei , = ^ l3ki, (3) 

M:+\x,)^J^~\l3,,,a,,), 

aij = -Jfjo'^] , Pij = -J,3a~-^jPi\j ■ (4) 

From the a and (3 values we can compute the estimated 
marginal distributions, which are Gaussian with mean fii = 

Ki{K + Y.keT, Pki) and variance K, = {Jn+Y^keTi "fei)~^- 
It is known that if GaBP converges, it results in the ex- 
act MAP estimate x*, although the variance estimates Ki 
computed by GaBP are only approximations to the correct 
variances [8]. 

2} Nonparametric belief propagation: In more general 
continuous-valued systems, the messages do not have a sim- 
ple closed form and must be approximated. Nonparametric 
belief propagation, or NBP, extends the popular class of par- 
ticle filtering algorithms, which assume variables are related 
by a Markov chain, to general graphs. In NBP, messages are 
represented by collections of weighted samples, smoothed by 
a Gaussian shape-in other words, Gaussian mixtures. 

NBP follows the same message update structure of (|2]i. No- 
tably, when the factors are all either Gaussian or mixtures of 
Gaussians, the messages will remain mixtures of Gaussians 
as well, since the product or marginalization of any mixture 
of Gaussians is also a mixture of Gaussians [3]. However, the 
product of d Gaussian mixtures, each with N components, 
produces a mixture of N'^ components; thus every message 
product creates an exponential increase in the size of the 
mixture. For this reason, one must approximate the mixture 
in some way. NBP typically relies on a stochastic sampling 
process to preserve only high-likelihood components, and a 
number of sampling algorithms have been designed to ensure 
that this process is as efficient as possible [9]-[ll]. One 
may also apply various deterministic algorithms to reduce the 
number of Gaussian mixture components [12]; for example, 
in [13], [14], an 0{N) greedy algorithm (where N is the 
number of components before reduction) is used to trade off 
representation size with approximation error under various 
measures. 

C. LDLC decoder 

The LDLC decoding algorithm is also described as a 
message-passing algorithm defined on a factor graph [6], 
whose factors represent the information and constraints on 
X arising from our knowledge of y and the fact that b is 
integer-valued. Here, we rewrite the LDLC decoder update 
rules in the more standard graphical models notation. The 
factor graph used is a bipartite graph with variables nodes 
{xi}, representing each element of the vector x, and factor 
nodes {fi,gs} corresponding to functions 

Hx.)=M{x,,y.,a^), g^xs) = |j ^;^j2e ' 

where Hg is the s*'' row of the decoding matrix H. Each 
variable node Xi is connected to those factors for which it is 
an argument; since H is sparse, Hg has few non-zero entries, 



making the resulting factor graph sparse as well. Notice 
that unlike the construction of [2], this formulation does not 
require that H be square, and it may have arbitrary entries, 
rather than being restricted to a Latin square construction. 
Sparsity is preferred, both for computational efficiency and 
because belief propagation is typically more well behaved 
on sparse systems with sufficiently long cycles [6]. We can 
now directly derive the belief propagation update equations as 
Gaussian mixture distributions, corresponding to an instance 
of the NBP algorithm. We suppress the iteration number r 
to reduce clutter. 

Variable to factor messages. Suppose that our factor to 
variable messages Msi{xi) are each described by a Gaussian 
mixture distribution, which we will write in both the moment 
and information form: 

(5) 

Then, the variable to factor message AIis{xs) is given by 

Mis{xs) = ^ TO LA/'(a-s ; mls,vL) = wl^J^^^jxs ; /3L,aL), 

(6) 

where 1 refers to a vector of indices [Ig] for each neighbor s, 

a\s =cr'^ + ^'ti . Pit = yicr"^ + ' ^'^^ 

I _ m^*; n wilf^-^x';P'^^ , all) 

J^'^x* -filial) 

The moment parameters are then given by f'j ~ {a\^)^^, 
m\f — /3jj(ajj)~^. The value x* is any arbitrarily chosen 
point, often taken to be the mean m\f for numerical reasons. 

Factor to variable messages. Assume that the incoming 
messages are of the form and note that the factor 
gs{-) can be rewritten in a summation form, gs{xs) = 
5{HsX = bg), which includes all possible integer values 
bs ■ If we condition on the value of both the integer bg and the 
indices of the incoming messages, again formed into a vector 
1 = [Ij] with an element for each variable j, we can see that 
gs enforces the linear equality HgiXi = bg — J^ ^sjXj. Using 
standard Gaussian identities in the moment parameterization 
and summing over all possible 6^ G Z and 1, we obtain 

Mst{xi) = ^ ^ w'iA/'(xi ; mli, vli) = 

bs 1 

y^,y^,wliAf~'^{xi ; (3li,ali) , (8) 

bs 1 

where 

iers\i 

mli = H~^{-bs + Y ^os-m-^), w\i = ]^ w]'^ , (9) 
jer^i jeTs\i 

and the information parameters are given by = (i^'i)^^ 
and/3^=m^(z.i,;)-i. 

Notice that (O matches the initial assumption of a Gaus- 
sian mixture given in (|5]l. At each iteration, the exact 
messages remain mixtures of Gaussians, and the algorithm 
iteslf corresponds to an instance of NBP. As in any NBP 
implementation, we also see that the number of components 



is increasing at each iteration and must eventually approxi- 
mate the messages using some finite number of components. 
To date the work on LDLC decoders has focused on de- 
terministic approximations [2], [15]-[17], often greedy in 
nature. However, the existing literature on NBP contains a 
large number of deterministic and stochastic approximation 
algorithms [9]-[13]. These algorithms can use spatial data 
structures such as KD-Trees to improve efficiency and avoid 
the pitfalls that come with greedy optimization. 

Estimating the codewords. The original codeword x can be 
estimated using its belief, an approximation to its marginal 
distribution given the constraints and observations: 

B,{xi) = U{x,) Y[ Msi{x^) . (10) 

The value of each Xi can then be estimated as either the 
mean or mode of the belief, e.g., x* = argmaxi3i(xj), 
and the integer-valued information vector estimated as b* = 
round(i7.T*). 

III. A Pairwise Construction of the LDLC 

DECODER 

Before introducing our novel lattice code construction, we 
demonstrate that the LDLC decoder can be equivalently con- 
structed using a pairwise graphical model. This construction 
will have important consequences when relating the LDLC 
decoder to Gaussian belief propagation (Section lIV-Bb and 
understanding convergence properties (Section 0. 

Theorem 1: The LDLC decoder algorithm is an instance 
of the NBP algorithm executed on the following pairwise 
graphical model. Denote the number LDLC variable nodes as 
n and the number of check nodes as }^ We construct a new 
graphical model with n + k variables, X = {xi, ■ ■ ■ , Xn+k) 
as follows. To match the LDLC notation we use the index 
letters i, j, .. to denote variables 1, n and the letters s, t, ... 
to denote new variables n + 1, ...,n + k which will take the 
place of the check node factors in the original formulation. 
We further define the self and edge potentials: 

oo 

^lji{xi) X J\f{xi;yi,a^) , tps{xs) = ^ A/'(xs ; 6^ , 0) , 

bg — — oc 

tpi^sixi,Xs) = exp{-XiHisXs) . (11) 
Proof: The proof is constructed by substituting the edge 
and self potentials ( fTSl ) into the belief propagation update 
rules. Since we are using a pairwise graphical model, we do 
not have two update rules from variable to factors and from 
factors to variables. However, to recover the LDLC update 
rules, we make the artificial distinction between the variable 
and factor nodes, where the nodes Xi will be shown to be 
related to the variable nodes in the LDLC decoder, and the 
nodes Xg will be shown to be related to the factor nodes in 
the LDLC decoder. 

a) LDLC variable to factor nodes: We start with the 
integral-product rule computed in the Xi nodes: 




'Our construction extends the square parity check matrix assumption to 
the general case. 



The product of a mixture of Gaussians fl A^t^(a;i) is itself 

teTi\s 

a mixture of Gaussians, where each component in the output 
mixture is the product of a single Gaussians selected from 
each input mixture Mti{xi). 

Lemma 2 (Gaussian product): [18, Claim 10], [2, Claim 
2] Given p Gaussians A/'(mi, wi), • • • ,N{mp, Vp) their prod- 
uct is proportional to a Gaussian A/'(m, v) with 

p ^ p p p 

i— 1 i — 1 1 1—1 

Proof: Is given in [18, Claim 10]. ■ 
Using the Gaussian product lemma the Ig mixture component 
in the message from variable node i to factor node s is a 
single Gaussian given by 



I dxi 



) dxi 



M\1{x^)= j ^i^{xi,Xs){Tpi{xi) Y\ M^-{xi)^c 
J. *eri\s 

ipis(xi,Xs){ipiixi)cxp{-^x^^( a[l) + Xi{Y 

J il^isi^i^ ^s) cxp(— + Xiyicr^'^)- 

■Gxp{-ia;;( ^ a\l)+Xi( ^ I3[l)]'^dxi = 
ter;\3 teTi\3 
I V'is(2:i, 3:s)(cxp{-ia;^((T^^ +^ a\\) + Xi(yi(j^'^ +^ /jjj ) da;; . 

We got a formulation which is equivalent to LDLC variable 
nodes update rule given in (|7]i. Now we use the following 
lemma for computing the integral; 

Lemma 3 (Gaussian integral): Given a (one dimensional) 
Gaussian 4)i{xi) oc M{xi;m,v), the integral 
/ ilji^s{xi,Xs)(f>i{xi)dxi, where is a (two dimensional) 

Xi 

Gaussian ipi^s{xi,Xs) = exp(— XiHi^Xs) is proportional to 
a (one dimensional) Gaussian ^/'^{Hism, Hf^v). 



Proof: 



'ipij{xi,Xj)(j)i{xi)dxi 



(X J exp{—XiHisXs)exp{ — ^{xi - m)^ /v}dxi 



exp 



(^(-^Xi/v) + (m/v - HisXs)x^dxi 



oc exp((m/v - HisXsf /{-I)) , 

where the last transition was obtained by using the Gaussian 
integral: 



oc 

/ 



exp {—ax^ + hx)dx — \J-k ja exp (6^/4a) 



exp ((m/i; — HisXsf K-"^)) = exp{-i(D(m/u - HisXs)^)} = 
= exp{-^(Hfsv)x'i + {Htsm)xs - \v{'m/v)'^} 
oc eicp{~\{Hisv)xi + {Hism)Xs} ■ 

■ 

Using the results of Lemma [3] we get that the sent message 
between variable node to a factor node is a mixture of 
Gaussians, where each Gaussian component k is given by 

Note that in the LDLC terminology the integral operation 
as defined in Lemma [3] is called stretching. In the LDLC 
algorithm, the stretching is computed by the factor node as 
it receives the message from the variable node. In NBP, the 
integral operation is computed at the variable nodes. 



LDLC Factors to variable nodes: We start again with 
the BP integral-product rule and handle the Xg variables 
computed at the factor nodes. 



Msi{Xi)= j ^i)is{Xi,Xs)'ll)s[Xs) 



dXs 



Note that the product J| MJ^ (xj ) , is a mixture of Gaus- 

sians, where the A;-th component is computed by selecting 
a single Gaussian from each message MJ^ from the set 
j G \ i and applying the product lemma (Lemma |2]i. We 
get 



Jijis{xi,Xs)(iJs{xs)exp{-^xl{ ^ h1^v[ 



+ 



fcera\i 



+Xs(^ 5Z -^'=s'^fes) i)'^^" (12) 
fcers\i 

We continue by computing the product with the self potential 

ipsixs) to get 

= J ■ipis{xi,X3){ ^cxpibsX^) cxp{- ^xI{Yh1^v[!J + 



+Xs(Y HkBin^J})dXs = 



-Y^ / 4>iB{xi,Xs){c-xp{b3X3)o-xp{-^xl{Y^L'"ks) + 



Y h^s(x^,x,){oxp{-^xUJ2HLv[\) + 



fcersXi 



Xs{bs +'Y, t^ksniks)})dxs = 



fceraXi 



Y / ^isi^i--^s){0Xp{-^xl(Y,^kaVks) + 

+xs{-bs + Y,Htsm[!J })dx, . 



Finally we use Lemma [3] to compute the integral and get 

+x,Hs,{ J2 Hlv[^J-\-b,+ Y Hk,ni['J}dx,. 
kers\i kers\i 

It is easy to verify this formulation is identical to the LDLC 
update rules Q. ■ 

IV. Using Sparse Generator Matrices 

We propose a new family of LDLC codes where the 
generator matrix G is sparse, in contrast to the original 
LDLC codes where the parity check matrix H is sparse. 
Table J] outlines the properties of our proposed decoder. Our 
decoder is designed to be more efficient than the original 
LDLC decoder, since as we will soon show, both encoding, 
initialization and final operations are more efficient in the 
NBP decoder. We are currently in the process of fully 
evaluating our decoder performance relative to the LDLC 
decoder. Initial results are reported in Section VI. 




Fig. 1. The approximating function gl^''^°'^(x) for the binary case. 



A. The NBP decoder 

We use an undirected bipartite graph, with variables nodes 
{hi\, representing each element of the vector h, and observa- 
tion nodes [zi] for each element of the observation vector y. 
We define the self potentials ipi{zi) and ^s{bs) as follows: 



ipi{zi) oc Af{zi;yi,( 



1 bs 

otherwise 



(13) 



and the edge potentials: 

ipi^s{zi,bs) = exp{—ZiGisbs) . 

Each variable node bg is connected to the observation nodes 
as defined by the encoding matrix G. Since G is sparse, 
the resulting bipartite graph sparse as well. As with LDPC 
decoders [7], the belief propagation or sum-product algo- 
rithm [6], [19] provides a powerful approximate decoding 
scheme. 

For computing the MAP assignment of the transmitted 
vector b using non-parametric belief propagation we perform 
the following relaxation, which is one of the main novel con- 
tributions of this paper. Recall that in the original problem, b 
are only allowed to be integers. We relax the function tps{xs) 
from a delta function to a mixture of Gaussians centered 
around integers. 



The variance parameter v controls the approximation quality, 
as ii — > the approximation quality is higher Figure 2 plots 
an example relaxation of ipi{bs) in the binary case. We have 
defined the self and edge potentials which are the input the 
to the NBP algorithm. Now it is possible to run the NBP 
algorithm using ^ and get an approximate MAP solution 
to (dJ. The derivation of the NBP decoder update rules is 
similar to the one done for the LDLC decoder, thus omitted. 
However, there are several important differences that should 
be addressed. We start by analyzing the algorithm efficiency. 

We assume that the input to our decoder is the sparse 
matrix G, there is no need in computing the encoding matrix 
G = H^^ as done in the LDLC decoder Naively this 
initialization takes 0{n^) cost. The encoding in our scheme 
is done as in LDLC by computing the multiplication Gb. 
However, since G is sparse in our case, encoding cost is 
0{nd) where d << ?i is the average number of non-zeros 
entries on each row. Encoding in the LDLC method is done in 



Algorithm 




IN or 


Tn iti QM'ynfirMi r\nf*Tnti r\n 
±lliLlallZ,u.LllJII UUCl clllUll 


G = 




Initialization cost 


0{n^) 




Encoding operation 


Gb 


Gb 


Encoding cost 


0{n2) 


0{nd), d<^n 


Post run operation 


Hx 


None 


Post run cost 


0{nd) 





TABLE I 

Comparison of LDLC decoder vs. NBP decoder 



Algorithm 


LDLC decoder 


NBP decoder 


Update rules 


Two 


One 


Sparsity assumption 


Decoding mat. H 


Encoding mat. G 


Algorithm derivation 


Custom 


Standard NBP 


Graphical model 


Factor graph 


Pairwise potentials 


Related Operations 


Stretch/Unstretch 


Integral 




Convolution 


product 




periodic extension 


product 



TABLE II 

INHERET DIFFERENCES BETWEEN LDLC AND NBP DECODERS 



0{v?) since even if H is sparse, G is typically dense. After 
convergence, the LDLC decoder multiplies by the matrix H 
and rounds the result to get b. This operation costs 0{nd) 
where d is the average number of non-zero entries in H. In 
contrast, in the NBP decoder, b is computed directly in the 
variable nodes. 

Besides of efficiency, there are several inherent differences 
between the two algorithms. Summary of the differences 
is given in Table We use a standard formulation of 
BP using pairwise potentials form, which means there is a 
single update rule, and not two update rules from left to 
right and right to left. We have shown that the convolution 
operation in the LDLC decoder relates to product step of the 
BP algorithm. The stretch/unstrech operations in the LDLC 
decoder are implemented using the integral step of the BP 
algorithm. The periodic extension operation in the LDLC 
decoder is incorporated into our decoder algorithm using the 
self potentials. 

B. The relation of the NBP decoder to GaBP 

In this section we show that simplified version of the NBP 
decoder coincides with the GaBP algorithm. The simphfied 
version is obtained, when instead of using our proposed 
Gaussian mixture prior, we initialize the NBP algorithm with 
a prior composed of a single Gaussian. 

Theorem 4: By initializing i/jsibg) ~ A/'(0, 1) to be a 
(single) Gaussian the NBP decoder update rules are identical 
to update rules of the GaBP algorithm. 

Lemma 5: By initializing ips{xs) to be a (single) Gaussian 

the messages of the NBP decoder are single Gaussians. 

Proof: Assume both the self potentials i}}s{bs),i>i{zi) 

are initialized to a single Gaussian, every message of the NBP 

decoder algorithm will remain a Gaussian. This is because 

the product (O of single Gaussians is a single Gaussian, 

the integral and ^ of single Gaussians produce a single 

Gaussian as well. ■ 

Now we are able to prove Theorem 4: 

Proof: We start writing the update rules of the variable nodes. 



We initialize the self potentials of the variable nodes ipii^i) 
JV(zi;yi,(j'^)^Now we substitute, using the product lemma 
and Lemma r" 



/ 'Ai,s(2!i, bs)( CXp(— i^^O-^^ +?/iZi(T^^) Y[ OXp( — i Q t i + 2; ft i ) ) dz; 

/'Vi,,(z,,be)(oxp(-iz;'(<T-^+ ati)+z,{a-^y.+ J2 Pti))dz, = 



oc cxp \ — -kz 



teri\s 



ferj\s 



Now we get GaBP update rules by substituting Ju = 

cr~2, Jis = Gis,hs = a-~'^yt ■■ 

Otis = -Jfsa'^\s ^ —Jis{Jii + ^ Qti)""^i 

teri\3 

Pis ^ -JisO^~^^sPi\s = -Jis^a'^^ihi + ^ AO) ■ 

teri\s 

We continue expanding 

i feer.v 
Similarly using the initializations 

/'Vi.s(z.,f'.)(cxp{-i6^} n cxp(-i()^Q„ + (),/3fc,))d6, = 
/''<A,,»(z.,f-,)(oxp{-ib^(l+ 2^ a„) + b,( J2 /3fc=)})d6,= 
exp{-ib^G?,(l + E a„)-l+b,G„(l+ ft,)} 

Now we get GaBP update rules by substituting Ja ^ 1, 

J si — Gisj • 

Osi = —JsiCtJ^i = —JsiiJii + ^ 0!is)~^, 

kers\i 

Pst = -JstaJ^if3s\i = -JBi(^Oi^\iihi + ^ /3fes)^ . 

fesrs\i B 

Tying together the results, in the case of a single Gaussian 
self potential, the NBP decoder is initialized using the 
following inverse covariance matrix: 

G 



We have shown that a simpler version of the NBP decoder, 
when the self potentials are initialized to be single Gaussians 
boils down to GaBP algorithm. It is known [20] that the 
GaBP algorithm solves the following least square problem 
minbgK" \\Gb — y\\ assuming a Gaussian prior on b, p{b) ~ 
7V(0, 1), we get the MMSE solution b* = {G'^Gy^G'^y. 
Note the relation to ([TJ. The difference is that we relax the 
LDLC decoder assumption that b e Z", with b £ R". 

Getting back to the NBP decoder. Figure 2 compares the 
two different priors used, in the NBP decoder and in the 
GaBP algorithm, for the bipolar case. It is clear that the 
Gaussian prior assumption on b is not accurate enough. In the 



NBP decoder, we relax the delta function il3[ to a Gaussian 
mixture prior composed of mixtures centered around Integers. 
Overall, the NBP decoder algorithm can be thought of as an 
extension of the GaBP algorithm with more accurate priors. 
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Fig. 2. Comparing GaBP prior to the prior we use in the NBP decoder for 
the bipolar case {b G { — 1, !})■ 

V. Convergence analysis 

The behavior of the belief propagation algorithm has been 
extensively studied in the literature, resulting in sufficient 
conditions for convergence in the discrete case [21] and in 
jointly Gaussian models [22]. However, little is known about 
the behavior of BP in more general continuous systems. 
The original LDLC paper [2] gives some characterization 
of its convergence properties under several simplifying as- 
sumptions. Relaxing some of these assumptions and using 
our pairwise factor formulation, we show that the conditions 
for GaBP convergence can also be applied to yield new 
convergence properties for the LDLC decoder. 

The most important assumption made in the LDLC con- 
vergence analysis [2] is that the system converges to a set 
of "consistent" Gaussians; specifically, that at all iterations r 
beyond some number tq, only a single integer bg contributes 
to the Gaussian mixture. Notionally, this corresponds to the 
idea that the decoded information values themselves are well 
resolved, and the convergence being analyzed is with respect 
to the transmitted bits Xj. Under this (potentially strong) 
assumption, sufficient conditions are given for the decoder's 
convergence. The authors also assume that H consists of a 
Latin square in which each row and column contain some 
permutation of the scalar values hi > ... > hd, up to an 
arbitrary sign. 

Four conditions are given which should all hold to ensure 
convergence: 

. LDLC-I: det{H) = dct(G) = 1. 
. LDLC-II: a < 1, where a = 

• LDLC-III: The spectral radius of p{F) < 1 where F is 
a n X n matrix defined by: 

if k ^ I and there exist a row r of H 
for which \Hri \ ~ hi and Hrk ^ 
otherwise 

• LDLC-IV: The spectral radius of piji) < 1 where H is 
derived from H by permuting the rows such that the hi 
elements will be placed on the diagonal, dividing each 



row by the appropriate diagonal element (+hi or —hi), 
and then nullifying the diagonal. 
Using our new results we are now able to provide new 
convergence conditions for the LDLC decoder. 

Corollary 6: The convergence of the LDLC decoder de- 
pends on the properties of the following matrix: 

'Hh- diaga/.^)) 
Proof: In Theorem [T] we have shown an equivalence 
between the LDLC algorithm to NBP initialized with the 
following potentials: 

oo 

i)i{xi) QC Af(xi;yi,a^) , i^sixs) = ^ J^~^ (xs;bs,0) , 

6s — — OO 

i>i,s{xi,Xs) = exp{xiHisXs) . (15) 

We have further discussed the relation between the self po- 
tential ^s{xs) and the periodic extension operation. We have 
also shown in Theorem |4] that if tpsixg) is a single Gaussian 
(equivalent to the assumption of "consistent" behavior), the 
distribution is jointly Gaussian and rather than NBP (with 
Gaussian mixture messages), we obtain GaBP (with Gaussian 
messages). Convergence of the GaBP algorithm is dependent 
on the inverse covariance matrix J and not on the shift vector 
h. 

Now we are able to construct the appropriate inverse 
covariance matrix J based on the pairwise factors given in 
Theorem [T] The matrix J is a 2 x 2 block matrix, where 
the check variables Xg are assigned the upper rows and the 
original variables are assigned the lower rows. The entries can 
be read out from the quadratic terms of the potentials ( fTSl ), 
with the only non-zero entries corresponding to the pairs 
{xi,Xs) and self potentials {xi,Xi). 

■ 

Based on Corollary |6] we can characterize the convergence 
of the LDLC decoder, using the sufficient conditions for 
convergence of GaBP. Either one of the following two 
conditions are sufficient for convergence: 

[GaBP-I] (walk-summability [22]) 

p{I - \D-^^'^JD-^/^\) < 1 where D = diag(J). 

[GaBP-II] (diagonal dominance [8]) J is diagonally 

dominant (i.e. | J,;i| >= Y^^^i \ Jij\,'^i)- 
A further difficulty arises from the fact that the upper 
diagonal of (fl4l i is zero, which means that both [GaBP-1,11] 
fail to hold. There are three possible ways to overcome this. 

1) Create an approximation to the original problem by 
setting the upper left block matrix of (fl4] | to diag(e) 
where e > is a small constant. The accuracy of the 
approximation grows as e is smaller In case either of 
[GaBP-1,11] holds on the fixed matrix the "consistent 
Gaussians" converge into an approximated solution. 

2) In case a permutation on J (O exists where either 
[GaBPI,II] hold for permuted matrix, then the "consis- 
tent Gaussians" convergence to the correct solution. 

3) Use preconditioning to create a new graphical 
model where the edge potentials are determined 
by the information matrix HH^, ipi^s{xi,Xs) = 



exp{xi{HH'^}isXs) and the self potentials of the Xi 
nodes are ipilxi) = exp{ — cr~^ + Xi{Hy}i}. The 
proof of the correctness of the above construction is 
given in [23]. The benefit of this preconditioning is 
that the main diagonal of HH^ is surely non zero. 
If either [GaBP-1,11] holds on HH^ then "consistent 
Gaussians" convergence to the correct solution. How- 
ever, the matrix HH^ may not be sparse anymore, thus 
we pay in decoder efficiency. 

Overall, we have given two sufficient conditions for conver- 
gence, under the "consistent Gaussian" assumption for the 
means and variances of the LDLC decoder Our conditions 
are more general because of two reasons. First, we present 
a single sufficient condition instead of four that have to 
hold concurrently in the original LDLC work. Second, our 
convergence analysis does not assume Latin squares, not 
even square matrices and does not assume nothing about the 
sparsity of H. This extends the applicability of the LDLC 
decoder to other types of codes. Note that our convergence 
analysis relates to the mean and variances of the Gaussian 
mixture messages. A remaining open problem is the conver- 
gence of the amplitudes - the relative heights of the different 
consistent Gaussians. 

VI. Experimental results 

In this section we report preliminary experimental results 
of our NBP-based decoder. Our implementation is general 
and not restricted to the LDLC domain. Specifically, recent 
work by Baron et aJ. [5] had extensively tested our NBP 
implementation in the context of the related compressive 
sensing domain. Our Matlab code is available on the web 
on [24]. 

We have used a code lengths of n = 100, n = 1000, 
where the number of non zeros in each row and each column 
is d = 3. Unlike LDLC Latin squares which are formed 
using a generater sequence hi, we have selected the non- 
zeros entries of the sparse encoding matrix G randomly 
out of { — 1,1}. This construction further optimizes LDLC 
decoding, since bipolar entries avoids the integral compu- 
tation (stretch/unstrech operation). We have used bipolar 
signaling, b G { — 1,1}. We have calculated the maximal 
noise level (j'^ax using Poltyrev generalized definition for 
channel capacity using unrestricted power assumption [25]. 
For bipolar signaling (7^^^ = 4 i^dct{G)^ /2TTe. When 
applied to lattices, the generalized capacity implies that there 
exists a lattice G of high enough dimension n that enables 
transmission with arbitrary small error probability, if and 
only if (T^ < CTj^naa;. Figure |3] plots SER (symbol error rate) 
of the NBP decoder vs. the LDLC decoder for code length 
71 = 100, n = 1000. The x-axis represent the distance from 
capacity in dB as calculated using Poltyrov equation. As can 
be seen, our novel NBP decoder has better SER for n = 100 
for all noise levels. For n = 1000 we have better performance 
for high noise level, and comparable performance up to 0.3dB 
from LDLC for low noise levels. We are currently in the 
process of extending our implementation to support code 
lengths of up n = 100, 000. Initial performance results are 
very promising. 
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Fig. 3. NBP vs. LDLC decoder performance 

VII. Future work and open problems 

We have shown that the LDLC decoder is a variant of 
the NBP algorithm. This allowed us to use current research 
results from the non-parametric belief propagation domain, 
to extend the decoder applicability in several directions. 
First, we have extended algorithm applicability from Latin 
squares to full column rank matrices (possibly non-square). 
Second, We have extended the LDLC convergence analysis, 
by discovering simpler conditions for convergence. Third, we 
have presented a new family of LDLC which are based on 
sparse encoding matrices. 

We are currently working on an open source implementa- 
tion of the NBP based decoder, using an undirected graphical 
model, including a complete comparison of performance to 
the LDLC decoder. Another area of future work is to examine 
the practical performance of the efficient Gaussian mixture 
product sampling algorithms developed in the NBP domain 
to be applied for LDLC decoder. As little is known about 
the convergence of the NBP algorithm, we plan to continue 
examine its convergence in different settings. Finally, we plan 
to investigate the applicability of the recent convergence fix 
algorithm [26] for supporting decoding matrices where the 
sufficient conditions for convergence do not hold. 
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